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1 Introduction 

The goal of this article is twofold. First, it presents an application of the 
theory of invariant convex cones of Lie algebras to the study of unitary rep- 
resentations of Lie supergroups. Second, it provides an exposition of recent 
results of the second author on the classification of irreducible unitary repre- 
sentations of nilpotent Lie supergroups using the method of orbits. 

In relation to the first goal, it is shown that there is a close connection 
between unitary representations of Lie supergroups and dissipative unitary 
representations of Lie groups (in the sense of [NeOO]). It will be shown that for 
a large class of Lie supergroups the only irreducible unitary representations 
are highest weight modules in a suitable sense. This circle of ideas leads 
to explicit necessary conditions for determining when a Lie supergroup has 
faithful unitary representations. These necessary conditions are then used to 
analyze the situation for simple and semisimple Lie supergroups. 

Pertaining to the second goal, the main results in [SalO] are explained in a 
more reader friendly style. Complete proofs of the results are given in [SalO], 
and will not be repeated. However, wherever appropriate, ideas of the proofs 
are sketched. 
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2 Algebraic background 

We start by introdiicing the notation and stating several facts which are used 
in this article. The reader is assumed to be familiar with basics of the theory 
of superalgebras, and therefore this section is rather terse. For more detailed 
accounts of the subject the reader is referred to [Be87], [Ka77], or [Sch79]. 

Let § be an arbitrary associative unital ring. A possibly nonassociative 
S-algebra s is called a superalgebra if it is Z2-graded, i.e., s = % ©Sy where 
SiSj C Si+j. The degree of a homogeneous clement a € S is denoted by \a\. 

A superalgebra s is called supercommutative if 

a6= (-l)l"l-l''l6a 

for every two homogeneous elements a, 6 £ s. 

A Lie superalgebra is a superalgebra whose multiplication, usually called 
its superbracket, satisfies graded analogues of antisymmetry and the Jacobi 
identity. This means that if A, B, C are homogeneous elements of a Lie su- 
peralgebra, then 

[A,B] = 

and 

(-I)I^HCI q] + {-l)\B\-\^\[B, [C, A]] + (-l)I^HB|[c, [A, B]] = 0. 

Let K be a field and g be a Lie superalgebra over K. If f) is a Lie subsu- 
peralgebra of g then i^(t)) denotes the supercommutant of f) in g, i.e., 

^,([)) = {Xefl|[0,x] = {o}}. 

The center of g is the supercommutant of g in g and is denoted by S^{q). 
The universal enveloping algebra of g, which is defined in [Ka77, Sec. 1.1.3], 
is denoted by ^(g). The group of K-linear automorphisms of g is denoted 
by Aut(g). Finally, recall that the definitions of nilpotent and solvable Lie 
superalgebras are the same as the ones for Lie algebras (see [Ka77, Sec 1]). 



2.1 Centroid, derivations, and differential constants 

Let K be an arbitrary field and s be a finite dimensional superalgebra over 
K. The multiplication algebra of s, denoted by ^(s), is the associative uni- 
tal superalgebra over K which is generated by the elements and L^, of 

EndK(s), for all homogeneous x <E s, where 

^x{y) = xy and Ra;(y) = (— l)''*' '^'?/a: for every homogeneous y S 5. 
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As usual, the superbracket on EndK(s) is defined by 

[A,B]= AB - {^1)^^^-^^\BA 

for homogeneous elements A,B & EndK(s), and is then extended to EndK(5) 
by linearity. The centroid of 5, denoted by 'rf{s), is the supercommutant of 
^(5) in EndK(5), i.e, 

<r(s) = {A€ EndK(s) I [A, B]=0 for every B e ^(s) }. 

Obviously 'rf{s) is a unital associative superalgebra over K. If = s then 
^(5) is supercommutative (see [Ch95, Prop. 2.1] for a proof). 

If s 6 {0, 1}, a homogeneous derivation of degree s of s is an element 
D G EndK(s) such that for every two homogeneous elements a, 6 S s, 

D{ab) = D(a)6+ (-l)l"l"*oD(6). 

The subspacc of EndK(s) which is spanned by homogeneous derivations of s is 
a Lie superalgebra over K and is denoted by DerK(s). The ring of differential 
constants, denoted by ^{s), is the supercommutant of DerK(5) in "^(s). 

Suppose that s is simple, i.e., ^ {0} and s docs not have proper two- 
sided ideals. By Schur's Lemma every nonzero homogeneous element of 'rf{s) 
is invertible. Since is always a two-sided ideal, 5^ = 5 and therefore '^{s) 
is supercommutative. It follows that '^{s)i = {0}, ^(s)o is a field, and ^(5) 
is a subfield of '^{s)q containing K. 



2.2 Derivations of base extensions 

Let K be a field of characteristic zero and A(n, K) be the Grafimann super- 
algebra over K in n indeterminates, i.e., the associative unital superalgebra 
over K generated by odd elements ^1 , • . . , modulo the relations 

^i^j + CjCi = for every l<i,j <n. 

Let s be a superalgebra over K. The tensor product s (S)k A(n, K) is a 
superalgebra over K. Note that since A(n, K) is supercommutative, if s is a 
Lie superalgebra then so is s ®k A(n, K). 

It is proved in [Ch95, Prop. 7.1] that 

DerK(s (E)K A(n, K)) = DerK(5) (g)K A(n, K) + "^(s) Ok W(n, K) (1) 

where 

W(n,K) =DerK(A(n,K)). 
The right hand side of (1) acts on 5 (8)k A(n, K) via 
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and 

(T 0K -Da)(X 0k a) = (-l)l^^l-l^lT(X) ®K £'A(a). 

Note that the right hand side of (1) is indeed a direct sum of the two sum- 

mands. This follows from the fact that every element of "^(s) ®k W(n, K) 
vanishes on s (8)k lA(n,K)j whereas an element of DerK(s) (g>K A(n, K) which 
vanishes on s ®k lA(n,K) must be zero. 



2.3 Cartan subsuperalgebras 

Let K be a field of characteristic zero and g be a finite dimensional Lie 
superalgebra over K. A Lie subsuperalgcbra of g which is nilpotent and self 
normalizing is called a Cartan subsuperalgcbra. 

An important property of Cartan subsuperalgebras of q is that they are 
uniquely determined by their intersections with Qq. Our next goal is to state 
this fact more formally. 

For every subset Wq of Qq, let 

= {X eg] for every e if fc > then iid{W)''{X) = }. 

One can easily prove that ^('JVg) is indeed a subsuperalgcbra of g. The next 
proposition is stated in [Sch87, Prop. 1] (see also [PeSe94, Prop. 1]). 

Proposition 2.3.1. If i) = t)^ (B i)- is a Cartan subsuperalgcbra of g then 1)q 
is a Cartan subalgebra of g^. Conversely, if f)o is a Cartan subalgebra of g-^ 
then <^(f)o) is a Cartan subsuperalgcbra of g. The correspondence 

is a bijection between Cartan subalgebras of gQ and Cartan subsuperalgebras 

ofg- 



2.4 Compactly embedded subalgebras 

Let be a finite dimensional Lie superalgebra over R. The group Aut(0) 
is a (possibly disconnected) Lie subgroup of GL(0), the group of invertible 
elements of EndK(0). The subgroup of Aut(0) generated by e'^^^o) is denoted 
by lnn(0). 
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If f)Q is a Lie subalgebra of 0g then INNg(f)Q) denotes the closure in Aut(fl) 
of the subgroup generated by e^^^o) , When INNg(f)Q) is compact f)Q is said 
to be compactly embedded in q. 

Cartan subalgebras of gg which are compactly embedded in q are especially 
interesting because they yield root decompositions of the cornplcxification of 
Q. The next proposition states this fact formally. In the next proposition, let 
T denote the usual complex conjugation of elements of gp = g 0k C, i.e., 
t{X + iY) = X-iY for every X,Y G Q. 

Proposition 2.4.1. Let % be a Cartan subalgebra of Qq which is compactly 
embedded in g. Then the following statements hold. 

(i) i-Q is abelian. 

(ii) One can decompose ^ as 

S^=00^'" (2) 

where 
and 

gC,a = eg^\[H,X]= ia{H)X for every i? G % }. 

(iii) IfaeA then -a € A as well, and if X e g^'" then t(X) e g^'"". 

(iv) 0o = %®[%>0o]- 

Proof. The proof of [NeOO, Theorem VII. 2. 2] can be adapted to prove Parts 
(i), (ii), and (iii). Part (iv) can be proved using the fact that % = ^-(tp) 
(see [Bo05, Chap. VII]). □ 

More generally, if gg- has a Cartan subalgebra which is compactly embed- 
ded in g, then any Cartan subalgebra of g"' yields a root decomposition. This 
is the content of the following proposition. 

Proposition 2.4.2. Assume that gg has a Cartan subalgebra which is com- 
pactly embedded in g. If i)^ is an arbitrary Cartan subalgebra of g^ then (}^ 
is abelian and there exists a root decomposition of g^ associated to kf', i.e., 

where 

Zi(f)^) = {ae (f)g)* I g^'« ^ {0}} 

and 

gC>« = {X &g^ \ [H,X]= ia{H)X for every F G f)§ }. 
Moreover, A{^'^) = -A{\f-). 
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Proof. Let tg be a Cartan subalgebra of Qq which is compactly embedded in 
Q. Then t^: is a Cartan subalgebra of q^, and by Proposition 2.3.1 it corre- 
sponds to a Cartan subsuperalgebra t"' of q^. Proposition 2.4.1 implies that 
there is a root decomposition oiQ^ associated to t'^, and if A{'f-) denotes the 
corresponding set of roots then AiiF) = —A{i^). It is known that any two 
Cartan subalgebras of are conjugate under inner automorphisms of g^. 
Using Proposition 2.3.1 one can show that any two Cartan subsupcralgebras 
of are conjugate under the group of C-linear automorphisms of gener- 
ated by e^^^^^T>\ By eonjugacy, the root decomposition associated to t"' turns 
into one associated to f)^. □ 



2.5 Simple and semisimple Lie superalgebras 

The classification of finite dimensional complex simple Lie superalgebras and 
their real forms is known from [Ka77] and [Se83] . Every complex simple Lie 
superalgebra is isomorphic to one of the following types. 

(i) A Lie superalgebra of classical type, i.e., A(m|n) where m,n > 0, 
B(m|n) where m > and n > 0, C(n) where n > 1, D{m\n) where 
m > 1 and n > 0, G(3), F(4), D(2|l,a) where a e C\{0,-1}, P(n) 
where n > 1, or Q(n) where n > 1. 

(ii) A Lie superalgebra of Cartan type, i.e., W(n) where n > 3, S(n) where 
n > 4, S(n) where n is even and n > 4, or H(n) where n > 5. 

(iii) A complex simple Lie algebra. 

Let s be a finite dimensional real simple Lie superalgebra with nontrivial 
odd part, i.e., Sj ^ {0}. Since '^{s) is a finite dimensional field exension of 
M, we have "^(s) = M or "^(s) — C. If "^(s) = C, then s is a complex simple 
Lie superalgebra which is considered as a real Lie superalgebra. If '5^(s) — M, 
then s is a real form of the complex simple Lie superalgebra s C>5r C. The 
classification of these real forms is summarized in Table 1 at the end of this 
article. 

A Lie superalgebra is called semisimple if it has no nontrivial solvable ide- 
als. Semisimple Lie superalgebras are not necessarily direct sums of simple 
Lie superalgebras. In fact the striicture theory of semisimple Lie siiperalge- 
bras is rather complicated. The following statement can be obtained by a 
slight modification of the arguments in [Ch95] . 

Theorem 2.5.1. If a real Lie superalgebra g is semisimple then there exist 
real simple Lie superalgebras si,. . . ,Sk and nonnegative integers ni, . . . , rife 
such that 



k k 

(si(8)K« A(ni,Ki)) C g C (DerK,(si)(S)Ki A(ni,Ki)+Li(8)Ki W(ni,Ki)) 

i=l i=l 



Lie supergroups, unitary representations, and invariant cones 7 
where Kj = ^{Si) and Lj = "^(Sj) for every 1 <i < k. 



3 Geometric background 

Since we are interested in studying unitary representations from an analytic 
viewpoint, wc need to realize them as representations of Lie supergroups on 
Z2-graded Hilbert spaces. To this end, we first need to make precise what we 
mean by Lie supergroups. 

One can define Lie supergroups abstractly as group objects in the category 
of supermanifolds. To give sense to this definition, one needs to define the 
category of supermanifolds. It will be seen below that this can be done by 
means of sheaves and ringed spaces. 

Nevertheless, the above abstract definition of Lie supergroups is not well- 
suited for the study of unitary representations, and a more explicit description 
of Lie supergroups is necessary. The aim of this section is to explain the 
latter description, which is based on the notion of Harish-Chandra pairs, 
and to clarify the relation between Harish-Chandra pairs and the categorical 
definition of Lie supergroups. 

This section starts with a quick review of the theory of supermanifolds. 
The reader who is not familiar with the basics of this subject and is interested 
in further detail is referred to [DeMo99], [Ko77], [Lc80], [Ma88], and [Va04]. 

We remind the reader that in the study of unitary representations only 
the simple point of view of Harish-Chandra pairs will be used. Therefore the 
reader may also skip the review of supcrgcomctry and continue reading from 
Section 3.4, where Harish-Chandra pairs are introduced. 



3.1 Supermanifolds 

Let p and q be nonnegative integers, and let ^rp denote the sheaf of smooth 
real valued functions on MP. The smooth {p\q) -dimensional superspace MP^'^ is 

the ringed space {M^, ^g^piq) where i^^rpi, is the sheaf of smooth superfunctions 
in q odd coordinates. The latter statement simply means that for every open 
U CRP one has 

^Rpi,(C/) = ffMp{U) (8>M A(g,K) 

and the restriction maps of ^rpi, are obtained by base extensions of the 
restriction maps of ^rp . 

The ringed space {W, ^rpi,) is an object of the category Tops_aig of topo- 
logical spaces which are endowed with sheaves of associative unital supcral- 
gebras over R. li X = {Xo,ffx) and y = {y^ffy) are objects in Top^.gig 
then a morphism : X ^ y is & pair [ipo, V*) such that ^po : Xo ^ is a 
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continuous map and 

if* -.ffy^ {iPo)*^X 

is a morphism of sheaves of associative unital superalgebras over M, where 
{^o)*^x is the direct image ^ of i^x- 

An object of Topj.gig is called a supermanifold of dimension {p\q) if it is 
locally isomorphic to K^*!'. supermanifolds constitute objects of a full subcat- 
egory of Top5_3|g. 



3.2 Some basic constructions for supermanifolds 

U Ai = {Mo,^m) is a supermanifold of dimension {p\q) then the nilpo- 
tent sections of generate a sheaf of ideals ^m- Indeed the underlying 
space jM.o is an ordinary smooth manifold whose sheaf of smooth functions is 
^m/^m- One can also show that if = {Mo, ^m) and M = {Mo, Gh) 
are two supermanifolds and : — >■ A/" is a morphism then the map 
V?o : Mo Mo is smooth (see [Le80, Sec. 2.1.5]). 

Locally, &mI '^M is isomorphic to i^kp. Therefore, if [/ C Mo is an open 
set, then for every section / G &m{U) and every point m & U the value f{m) 
is well defined. In this fashion, from any section / one obtains a smooth map 

f:U^M. 

Nevertheless, because of the existence of nilpotent sections, / is not uniquely 
determined by /. 

Supermanifolds resemble ordinary manifolds in many ways. For example, 
one can prove the existence of finite direct products in the category of su- 
permanifolds. Moreover, for a supermanifold M of dimension {p\q) the sheaf 
DerR(if?'_vi) of R-linear derivations of the structural sheaf i^j^ is a locally free 
sheaf of ^_a4 -modules of rank {p\q). Sections of the latter sheaf are called vec- 
tor fi,elds of M . The space of vector fields is closed under the superbracket 
induced from EndR(^»). 

U M = {Mo, ^m) is a supermanifold and m G Mo, then there exists an 
obvious morphism 

(5™ : M°l° ^ A! 

where {5m) o : — >■ AIo maps the unique point of M° to m, and for every 
open set U C Mo if / G Gm{U) then 



^ In [DeMo99] the authors define morphisms based on puUback. Since puUback and direct 
image are adjoint functors, the definition of [DeMo99] is equivalent to the definition given 
in this article, which is also used in [Le80] . 
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Moreover, is a terminal object in the category of super manifolds. Indeed 
for every supermanifold A4 = {Mo, & m) there exists a morphism 

such that (kx)o : M.o maps every point ol Mo to the unique point of 

M° and for every t G ^ro|o(]R°) ~ M one has (KA<)#(i) = t-lM- 



3.3 Lie supergroups and their Lie superalgebras 

Recall that by a Lie supergroup we mean a group object in the category 
of supermanifolds. In other words, a supermanifold Q = {Go,ffg) is a Lie 
supergroup if there exist morphisms 

n-.g xG-^g, £ : R°i° -^g, and L-.g^g 

which satisfy the standard relations that describe associativity, existence of 
an identity element, and inversion. It follows that is a Lie group whose 
multiplication is given by Ho '■ Qo x Go ^ Go- 
To a Lie supergroup G one can associate a Lie superalgebra Lic(^) which 
is the subspace of DerR(^g) consisting of left invariant vector fields of G- The 
only subtle point in the definition of Lie(^) is the definition of left invariant 
vector fields. Left invariant vector fields can be defined in several ways. For 
example, in [DeMo99] the authors use the functor of points. We would like 
to mention a different method which is also described in [BoSaQl]. For every 
g &Go, one can define left translation morphisms Xg : G ^ G hy 

\g= jjLo [[5g o Kg) X idg) 

where idg : G ^ G is the identity morphism. Similarly, one can define right 
translation morphisms 

Pg = jJLO (idg X {5g o Kg)). 

A vector field D is called left invariant if it commutes with left translation, 
i.e., 

i\g)*0D = D0{\g)*. 

It is easily checked that Lie{G), the space of left invariant vector fields of G, 
is closed under the super bracket of DerR(^_A4)- Moreover, there is an action 
of Go on Lie(^) given by 



D^{Pg)*0D0{p,^^g^)*. 



(3) 
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Because of Part (ii) of Proposition 3.3.1 below it is natural to denote this 
action by Ad (5). 

Proposition 3.3.1. For a Lie supergroup Q = {Qo, Gg) the following state- 
ments hold. 

(i) Lie(S) = Lie{Q)Q © Lie{G)j is a Lie superalgebra over R. 

(ii) The action of Qo on Lie(0) given by (3) yields a smooth homomor- 
phism of Lie groups 

Ad-.Qo^ GL(Lie(g)) 

such that Ad(^o) C EndK(Lie(^))o. 

(iii) Lie{Q)Q is the Lie algebra of Qo and if d{ Ad) denotes the differential 
of the above map Ad, then 

d(Ad)(x)(y) = ad(x)(y) 

for every X e Lie(^)Q and every Y e Lie(^), where 

ad{X){Y) ^ [X,Y]. 



3.4 Harish- Chandra pairs 

Proposition 3.3.1 states that to a Lie supergroup Q one can associate an 
ordered pair {Qo,Lie{Q)), where Qo is a real Lie group and Lie(^) is a Lie 
superalgebra over M, which satisfy certain properties. Such an ordered pair 
is a Harish-Chandra pair. 

Definition 3.4.1. A Harish-Chandra pair is a pair {G,q) consisting of a Lie 
group G and a Lie superalgebra g which satisfy the following properties. 

(i) 0Q is the Lie algebra of G. 

(ii) G acts on g smoothly by K-lincar automorphisms. 

(iii) The differential of the action of G on g is equal to the adjoint action 
of Qq on Q. 

One can prove that 

Q^{QoM<Q)) 

is an equivalence of categories from the category of Lie supergroups to the 
category of Harish-Chandra pairs. Under this equivalence of categories, a 
morphism ^ : C/ — >• "H in the category of Lie supergroups corresponds to a 
pair (V'c'^i'Lie) where tjjo ■ Qo ^ "Ho is a homomorphism of Lie groups, 



^Lie : Ue{Q) Ue{n) 
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is a homomorphism of Lie superalgebras, and 



d'^o = V'Lie 



Lie(e)o" 



Remark 3.4-S. Using Harish-Chandra pairs one can study Lie supergroups 
and their representations without any reference to the structural sheaves. In 
the rest of this article, Lie supergroups will always be realized as Harish- 
Chandra pairs. 



4 Uniteiry representations 

According to [DeMo99, Sec. 4.4] one can define a finite dimensional super 
Hilbert space as a finite dimensional complex Z2 -graded vector space which 
is endowed with an even super Hermitian form. Nevertheless, since the even 
super Hermitian form is generally indefinite, in the infinite dimensional case 
one should address the issues of topological completeness and separability. 
For the purpose of studying unitary representations it would be slightly more 
convenient to take an equivalent approach which is more straightforward, but 
less canonical. 



4-1 Super Hilbert spaces 

A super Hilbert space is a Z2-graded complex Hilbert space Jif = Ji% © 
such that Jf% and are mutually orthogonal closed subspaces. If (■, ■) 

denotes the inner product of J^f, then for every two homogeneous elements 
v,w ^ the even super Hermitian form (f , w) of ,J^/f is defined by 



One can check that (•, ■) satisfies the properties stated in [DeMo99, Sec. 4.4]. 
In this article the latter sesquilinear form will not be used. 



4-2 The definition of a unitary representation 

In order to obtain an analytic theory of unitary representations of Lie super- 
groups one should deal with the same sort of analytic difficulties that exist 
in the case of Lie groups. One of the main difficulties is that in general one 




if i; and w have opposite parity, 
{v, w) if V and w are even, 
i{v,w) if w and w are odd. 



12 



Kajrl-Hermann Neeb and Hadi Salmasian 



cannot define the differential of an infinite dimensional representation of a Lie 

group on the entire representation space. However, one can always define the 
differential on certain invariant dense subspaces, such as the space of smooth 
vectors. 

In the rest of this article, the reader is assumed to be familiar with classical 
results in the theory of unitary representations of Lie groups. For a detailed 
and readable treatment of this subject see [Va99]. 

If ,y/f is a (possibly Z2-gradcd) complex Hilbcrt space, the group of unitary 
operators of Jf is denoted by \J{J^). As usual, if tt : G — >■ U(Jf ) is a 
unitary representation of a Lie group G, then the space of smooth vectors 
(respectively, analytic vectors) of (tt, Jf) is denoted by Jf°° (respectively. 

Definition 4.2.1. Let (G, g) be a Lie supergroup. A unitary representation 
of (G,0) is a triple {iT,p^,J^) satisfying the following properties. 

(i) = J% © is a super Hilbert space. 

(ii) (tt, Jf) is a unitary representation of G and Tr{g) G Endc(^)o ^'^^ 
every g G G. 

(iii) : Q ^ Endc{J^°°) is an M-linear Z2-graded map, where 
denotes the space of smooth vectors of (tt, Jf). Moreover, for every 

p^X)p^Y) + p^Y)p^X) = -ipmX,Y]). 

(iv) If dTT denotes the differential of tt then p"{X) = diT{X) for every 
X e Bo- 

(v) For every X G fly the operator p'^{X) is symmetric, i.e., if VjW € M"^ 
then 

{p-{X)v,w) = i,v,p\X)w). 

(vi) For every g gG and every X £ q, 

p^{Ad{g)iX))=n{g)p^XMg)-K 

Remark 4- 2- 2. It is easy to see that by letting an element X-^+Xj e % ®flT 
act on as 

p-{X^) + e^^p^{Xj) 

one obtains from p^ a homomorphism of Lie superalgebras from q into 
Endc(J^°°). 

Remark 4-2.3. Subrepresentations, irreducibility, and unitary equivalence of 
unitary representations of Lie supergroups are defined similar to unitary rep- 
resentations of Lie groups (see [CCTV06]). Note that in the definition of 
unitary equivalence, intertwining operators are assumed to preserve the grad- 
ing. This means that in general a unitary representation is not necessarily 
unitarily equivalent to the one obtained by parity change. 
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Lemma 4.2.4. For each X e g, the operator p^{X): J>f°° is con- 

tinuous with respect to the Frechet topology on Jf°°. Moreover, the bilinear 
map 

Q X M"^, {X, v) ^ p''{X)v (4) 

is continuous. 

Proof. Since q is finite dimensional, it suffices to show that each operator 
p'^{X) is continuous. For X & Qq , this follows from the definition of the 
Frechet topology on 

For X S Qj, the operator p^iX) on is symmetric (sec Definition 

4.2.1(v) ), hence the graph of p^{X) is closed. Now the Closed Graph Theorem 
for Frechet spaces (see [Ru73, Thm. 2.15]) implies its continuity. □ 

From now on we assume that ^ is separable. Although this assumption 
is not needed in Definition 4.2.1, it helps in avoiding technical conditions in 
various constructions, e.g., when induced representations are defined. Note 
that if (tt,/?'^,^) is irreducible then is separable. 

In Definition 4.2.1 the fact that is chosen as the space of the repre- 

sentation of Q is not a limitation. In fact it is shown in [CCTV06, Prop. 2] 
that in some sense any reasonable choice of the space of the representation 
of 0, i.e., one which is dense in and satisfies natural invariance proper- 
ties under the actions on G and q, would yield a definition equivalent to the 
one given above. This fact also plays a role in showing that restriction and 
induction functors are well defined. Another useful fact, which follows from 
[CCTV06, Prop. 3] , is that the space " of analytic vectors of (tt, ) is 
invariant under P^{q). 



4' 3 Restriction and induction 

Suppose that Q = {G,g) is a Lie supergroup, and T-L = {H, f}) is a Lie subsu- 
pergroup of G. Let (tt, p'" , Jf) be a unitary representation of G. A priori it is 
not clear how to restrict (tt, p^,Jf) to H. The difficulty is that in general the 
space of smooth vectors of the restriction of (tt, J^) to H will be larger than 
Jif°°. To circumvent this issue one can use [CCTV06, Prop. 2] to show that 
the action of H on °° determines a unique unitary representation of H on 
Jf. This representation is called the restriction of {iTjp^jJf) to H, and is 
denoted by 

Res^(7r,/9'', Jf). 

Inducing from 'H to ^ is more delicate. Let {a, p'^, JXf) be a unitary repre- 
sentation of H. The first step towards defining a representation (tt, p^ , J^) of 
G that is induced from (a, p'^ ,Jf) is to identify the super Hilbert space Jf. By 
analogy with the case of Lie groups one expects the super Hilbert space to 
be a space of ^-valued functions on G which satisfy an equivariance property 
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with respect to the left regular action of 'H. One can then describe the action 
of Q by formal relations, hoping that a unitary representation, as defined in 
Definition 4.2.1, is obtained. This formal approach leads to technical com- 
plications and it is not clear how to get around some of them. Nevertheless, 
at least in the special case that the homogeneous super space 'H\Q is purely 
even, i.e., when dimgj = dimfjj, it is shown in [CCTV06, Sec. 3] that the 
induced representation can be defined rigorously. In this article, only the spe- 
cial case when both G and H arc imimodular groups is used, and in this case 
the induced representation is defined as follows. Since the homogeneous space 
is purely even, there is a natural isomorphism ~ H\G. Choose an 
invariant measure /i on H\G, and let ^ be the space of measurable functions 
f : G which satisfy the following properties. 

(i) !{hg) = a{h)f{g) for every g G G and every h € H. 



and one can easily check that it is unitary with respect to the standard inner 
product of J^. The most natural way to define the action of an element 
X £ Qj on an element / € Jif°° is via the formula 



It is known that every / € °° is a smooth function from G to and 
fig) e J^°° for every g G G [Po72, Th. 5.1]. Consequently, the right hand 
side of (5) is well defined. However, a priori it is not obvious why for an 
element X G Qj the right hand side of (5) belongs to One can prove 

the weaker statement that p'^{X)f e using a trick which is based on the 
ideas used in [CCTV06]. Since this trick sheds some light on the situation, it 
may be worthwhile to mention it. One can prove that the operator p^{X) is 
essentially self-adjoint. Let p^{X) denote the closure of p^{X). The operator 

2 

I + p'^{X) has a bounded inverse whose domain is all of (this follows for 
instance from [Co85, Chap. X, Prop. 4.2]). For every / € 



Using the spectral theory of self-adjoint operators one can show that the 
operator p'^{X){I + p'^{X) ) ^ is bounded. Moreover, 




(7r(,g)./)(.gi) = /(gi.g) for every g,gi&G, 



{p-{X)f){g) = p'^{kd{g){X)){f{g)). 



(5) 



p^{x)f = p-{x)f = p-{x){i + p-{xf)-\i + p-{xf)f. 



{I + p^{xf)f={. 



;j-^d7rax,x]))/G j^' 



Finally, boundedness of p'^{X){I + p'^(X) implies that p"{X)f e . 
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To prove that indeed p^{X)f e requires more effort. This is proved 
in [CCTV06, Sec. 3] in an indirect way. The idea of the proof is to find a dense 
subspace ^ C such that p'^ie)^ C £g. As shown in [CCTV06, Sec. 3], 
one can take ^ to be the subspace of °° consisting of functions from G 
to with compact support modulo H. That (tt, p'^, J^) is well defined then 
follows from [CCTV06, Prop. 2]. 

The representation (tt, p'^, Jf) induced from [(j,p'' ,Jf) is denoted by 



It can be shown [SalO, Prop. 3.2.1] that induction may be done in stages, 
i.e., if is a Lie subsupergroup of Q, K, is & Lie subsupergroup of %, and 
{(j,p'^ ,J(f) is a unitary representation of /C, then 



5 Invariant cones in Lie algebras 

The goal of this section is to take a brief look at convex cones in finite 
dimensional real Lie algebras which are invariant under the adjoint action. A 
natural reduction to the case where the cone is pointed and generating leads 
to an interesting class of Lie algebras with a particular structure that will be 
discussed below. 

A closed convex cone C in a finite dimensional vector space V is said to 
be pointed if C fl — C = {0}, i.e., if C contains no afhne lines. It is said to be 
generating if C — C = or equivalently if Int{C) is nonempty, where Int{C) 
denotes the set of interior points of C. If C is a cone in a finite dimensional 
vector space V then C* denotes the cone in V* consisting of all X&V* such 
that A(a;) > for every x G C. 



5.1 Pointed generating invariant cones 

Let g be a finite dimensional Lie algebra over R. A cone C C g is called 
invariant if it is closed, convex, and invariant under Inn(fl). 

Suppose that C is an invariant cone in g and set H(C) = C fl —C and 
g(C) = C — C. The subspaces H(C) and g(C) are ideals of g and C/H(C) is a 
pointed generating invariant cone in the quotient Lie algebra 0(C) /H(C) . The 
main concern of the theory of invariant cones is to understand the situation 
when C is pointed and generating. 

The existence of pointed generating invariant cones in a Lie algebra has 
the following simple but useful consequence. 



Ind?, 



Ind^Ind^(<7,p^ JT) ~ Ind|(a,p-, 
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Lemma 5.1.1. Let C be a pointed generating invariant cone in q. If a is an 
abelian ideal of q then a C 3f{Q). 

Proof If X e lnt{C), then C D e-'^^'^^X = X + [a,X]. Since C contains no 
affine lines, [a,X] = {0}. Since X e lnt{C) is arbitrary, a C ^(fl). □ 

To study invariant cones further, we need the foUowing lemma. 

Lemma 5.1.2. Let V be a finite dimensional vector space, S QV be a convex 
subset, and K C GL{y) be a subgroup which leaves S invariant. Suppose that 
the closure of K in GL(y) is compact. If S is open or closed, then it contains 
K -fixed points. 

Proof. Let K be the closure of K, and be a normalized Haar measure 
on K. For every v £ S, the point Vo = //f • v) diJj^{k) is K-fixed, and it is 
easily verified that Vo & S. □ 

The preceding lemma has the following interesting consequence for invari- 
ant cones. 

Lemma 5.1.3. Let C C q be a pointed generating invariant cone. Then a 
subalgebra t C q is compactly embedded in q if and only if i2g (t) fl lnt{C) ^ 0. 

Proof. If 6 C is compactly embedded in q then Lemma 5.1.2 implies that 
lnt{C) contains fixed points for Inng(fi), i.e., 

2fgit) n intiC) 9. (6) 

Conversely, if ^(J) fl lnt{C) ^ then set K = Inng(t) and observe that K 
is a subgroup of lnn(0) with a fixed point Xq € int{C). The set C fl {Xq — C) 
is a compact i^-invariant subset of g with interior points. This implies that 
K is bounded in GL(g) and therefore it has compact closure in Aut(0). □ 



5.2 Compactly embedded Cartan subalgebras 

Let be a finite dimensional Lie algebra over M. Our next goal is to show 
that the existence of a pointed generating invariant cone in q implies that 

g has compactly embedded Cartan subalgebras. The next lemma shows how 
such a Cartan subalgebra can be obtained explicitly. 

Lemma 5.2.1. Let C C g be a pointed generating invariant cone. Suppose 
that Y e int{C) is a regular element of g, i.e., the subspace 

^g(y) = UWad(rr) 

n 

has minimal dimension. If i = ker(ad(y)), then i is a Cartan subalgebra of 
g which is compactly embedded in g. 
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Proof. For any such Y, the subspace i = ^g{Y) is a Cartan subalgebra of 
(see [Bo05, Chap. VII]). Since Y e Sfgi^giRY)), Lemma 5.1.3 impHes 
that ^(RF) is compactly embedded in g. It follows immediately that WY 
is compactly embedded in g. Therefore the endomorphism ad(F) : g — >^ g is 
semisimple and 

t = ker(ad(r)) ^ 3fg{Y) 
from which it follows that t is compactly embedded in g. □ 

Remark 5.2.2. It is known that the set of regular elements of g is dense (see 
[Bo05, Chap. VII]). Since lnt{C) ^ 0, the intersection of lnt{C) with the set 
of regular elements of q is nonempty. 



5.3 Characterization of Lie algebras with invariant 
cones 

The material in this section is meant to shed light on the connection between 
invariant cones and Hermitian Lie algebras. The reader is assumed to be 
familiar with the classification of real semisimple Lie algebras. 

The study of invariant cones in finite-dimensional Lie algebras was initi- 
ated by B. Kostant, I. E. Segal and E. B. Vinberg [Se76], [VinSO]. A structure 
theory of invariant cones in general finite dimensional Lie algebras was de- 
veloped by Hilgcrt and Hofmann in [IIiHo89] . The characterization of those 
finite dimensional Lie algebras containing pointed generating invariant cones 
was obtained in [Ne94] in terms of certain symplectic modules called of convex 
type, whose classification can be found in [NcuOO]. A self-contained exposi- 
tion of this theory is available in [NeOO] , where the Lie algebras g for which 
there exist pointed generating invariant cones in g ® M are called admissible. 

Example 5.3.1. (cf. [VinSO]) Suppose that g is a real simple Lie algebra with 
a Cartan decomposition g = I © p. Since p is a simple nontrivial 4-module, 
^(£) = 3f{t). If C is a pointed generating invariant cone in g, then from 
Lemma 5.1.3 it follows that 

/»t(C)n^(e) ^0. 

In particular 2f{t) ^ {0}, i.e., g is HermAfAan. Conversely, assume that g is 
Hermitian and ^ Z £ iF(£). If (•, •) denotes the Killing form of g, then from 
the Cartan decomposition Inn(g) = Inn(6)e*'*^''^ it follows that 

(Inn(g)Z,Z) = (e^^(P)Z,Z). 



If P G p then {e^<^(P^Z, Z)<0 because 
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(ad(P)2"(Z),Z) 
(2n)! 



n=0 



and the linear transformations ad(P)^" : { — > 6 are positive definite with 
respect to (•,•). It follows that Inn(g)Z lies in a proper invariant cone C C q. 
Since g is simple, C is pointed and generating. 

A slight refinement of the above arguments shows that a reductive Lie 
algebra q is admissible if and only if ^g{^{t)) = £ holds for a maximal 
compactly embedded subalgebra I of g. Lie algebras satisfying this property 
arc called quasihermitian. This is equivalent to all simple ideals of g being 
either compact or Hermitian. A reductive admissible Lie algebra contains 
pointed generating invariant cones if and only if it is not compact semisimple. 
This clarifies the structure of reductive Lie algebras with invariant cones. 

Below we shall need the following lemma. 

Lemma 5.3.2. Let g be a quasihermitian Lie algebra, t C q be a maximal 
compactly embedded subalgebra of g, and and : g — > 2f{t) be the fixed point 
projection for the compact group e^'^'. Then every closed invariant convex 
subset C C g satisfies p^{C) = C n ^{i). 

Proof. Let p C g be a f-invariant complement and recall that g is said to be 
quasihermitian if i. = ^g{2f{t)). This condition implies in particular that p 
contains no non-zero trivial 8-submodule, so that ^(t) = The asser- 

tion now follows from the proof of Lemma 5.1.2. □ 

In the case of an arbitrary Lie algebra g having a pointed generating in- 
variant cone, one can use Lemma 5.1.1 to show that the maximal nilpotent 
ideal n of g is two-step nilpotent, i.e., a generalized Heisenberg algebra. More- 
over, n clearly contains i?°(g), which is contained in any compactly embedded 
Cartan subalgebra t of g. Let a C t be a complement to 3f{g) and s be a 
i-invariant Levi complement to n in g (which always exists), and set [ = a®s. 
Then I is reductive, g = ( k n, and [ is an admissible reductive Lie algebra (see 
[NeOO, Prop. VII. 1.9]). At this point the structure of n and I is quite clear. 
However, to derive a classification of Lie algebras with invariant cones from 
this scmidirect decomposition, one has to analyze the possibilities for the 
[-module structure on n in some detail. This is done in [Ne94] and [NeuOO]. 



6 UnitEiry representations and invariant cones 

A Lie supergroup Q = {G,g) is called -k-reduced if for every nonzero AT € g 
there exists a unitary representation {iT,p'^,'H) of Q such that p'^{X) ^ 0. 
Note that when g is simple, Q is ^-reduced if and only if it has a nontrivial 
unitary representation. In this section we study properties of ^-reduced Lie 
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supergroups via methods based on the theory of invariant cones. We obtain 

necessary conditions for a Lie supergroup Q to be ★-reduced. It turns out 
that these necessary conditions are strong enough for the classification of 
★-reduced simple Lie supergroups. 

Let Q = (G, g) be an arbitrary Lie supergroup, and let {'K,p",M') be a 
unitary representation of Q. Fix an element X G Qj. From 

p^{[X,X]) = i[p^X),p^{X)] = 2ip^Xf 

and the fact that the operator p^{X) is symmetric it follows that 

{ip^{[X, X])v, v)<Q for every v G 

Let Cone{Q) denote the invariant cone in flg which is generated by elements 
of the form [X, X\ where X G Qj. Linearity of p" implies that 

{ip^{Y)v, v) <0 for every v G M"^ and every Y G Cone{g). (7) 

This means that tt is Cf»e(5)-dissipative in the sense of [NeOO]. 



6.1 Properties of -k -reduced Lie supergroups 

Unlike Lie groups, which are known to have faithful unitary representations, 
certain Lie supergroups do not have such representations. The next proposi- 
tion, which is given in [SalO, Lcm. 4.1.1], shows how this can happen. The 
proof of this proposition is based on the fact that for every X G 0i, the 
spectrum of —ip'"{[X,X]) is nonnegative, so that a sum of such operators 
vanishes if and only if all summands vanish. 

Proposition 6.1.1. Let (tt, p'^ , J^) he a unitary representation ofQ = {G, q). 
Suppose that elements X^,. . . , X^ G Qj satisfy 

[Xi,X,] + --- + [Xm,Xm] = 0. 

Then p-^iXi) = ■■■ = p"{Xm) = 0. 

The next proposition provides necessary conditions for a Lie supergroup 
to be ★-reduced. 

Proposition 6.1.2. IfQ = {G,q) is -k-reduced, then the following statements 
hold. 

(i) Conc{Q) is pointed. 

(ii) For every A G Int{Cone{Q)*) , the symmetric bilinear form 



^^A : 00 X 00 ^ » defined by Qx{X, Y) = X{[X, Y]) 
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is positive definite . 

(iii) Let t-^ be a Lie subalgebra of flg. // 1-^ is compactly embedded in Qq, 
then {g is compactly embedded in q as well. 

(iv) If Qq = [g-, Q-] then Qq has a Cartan subalgebra which is compactly 

emhedded, in g. 

(v) Assume that there exists a Cartan subalgebra fig- of Qq which is com- 
pactly embedded in g. Let p : — > [)q 6e the projection map corre- 
sponding to the decomposition 

00 = f)o® [%'0o] 

(see Proposition 2.4-1) and p* : [}^ ^ be the corresponding dual 
map. Then 

Int{Coneigy) np*{i)^)^t 

Proof, (i) Suppose, on the contrary, that Y, —Y G Cone{Q) for some nonzero 
Y. Let (tt, jJt') be a unitary representation of Q. For every v € J^°°, 

< {ip''{Y)v,v) < 

which imphes that {ip'^{Y)v,v) = 0. Therefore for every v,w € J^°° and 
every z € C, 

= {{ip'^iY)) {v + zw), V + zw) 
^ {ip'^{Y)v, v) + z{ip''{Y)v, w) + z{ip''{Y)w, v) + \zf {ip" {Y)w, w) 
= z{ip''{Y)v, w) + z{ip''{Y)w, v) 

and since z is arbitrary, {ip^ (Y)v,w) = for every v.w G Jif°°. This means 
that p^{Y) = 0, hence Y = because Q is :*r-reduccd. 

(ii) That Q.\ is positive scmiclefinitc is immediate from the definition of 
Cone{g)*. If X e gi satisfies QxiX,X) = tlicn from A G lnt{Cone(g)*) it 
follows that [X, X] = 0. Since Q is ^-reduced. Proposition 6.1.1 implies that 

(iii) Part (i) implies that Cone(g) is pointed, and therefore lnt{Cone{g)*) 
is nonempty [NeOO, Prop. V.I. 5]. The action of the compact group INNg_({Q) 
on Conc{Q)* leaves lnt{Conc{Q)*) invariant. By Lemma 5.1.2, this action has 
a fixed point A € lnt{Cone{Q)*). Therefore the symmetric bilinear form Q.\ of 
Part (ii) is positive definite and invariant with respect to INNg(6g). From the 
inclusion Aut(g) C Aut(gQ) x GL(gY) it follows that INNg(4Q) is compact. 

(iv) By Part (iii) it is enough to prove the existence of a Cartan subal- 
gebra which is compactly embedded in g^. Part (i) implies that Com{Q) is 
pointed. The equality g^ = [g-jfll] means that Cone{g) is generating. There- 
fore Lemma 5.2.1 completes the proof. 

(v) Part (i) implies that lnt{Cone{Q)*) ^ 0. Since INNg_(f)g) is compact 
and leaves lnt{Cone{Q)*) invariant. Lemma 5.1.2 implies that there exists a 
fj, e lnt{Cone{Q)*) which is fixed by INNg_([)Q), i.e., contained in p*([}i). □ 
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Proposition 6.1.3. Suppose that Q = (G, g) is a -k-reduced Lie supergroup. 
Let 

(i) 1)q be a Cartan subalgebra of Qq which is compactly embedded in Q, 

(ii) A be the root system associated to ()o (see Proposition 2.4-1), 

(ill) fj, G int{cont{Q)*) np*(t)^), where p* is the map defined in the state- 
ment of Proposition 6.1.2. 

Then for every nonzero a €: A the Hermitian form 

(■,-)a:0Y ^% 

defined by {X,Y)a = I2{[X,Y]) is positive definite. 
Proof. Let 

be the symmetric bilinear form defined by 

n^{X,Y) = i,{[X,Y]). 

By Proposition 6.1.2(ii) tlie form is positive definite. If X e Qj'" then 
X G gj'~" and 

Q^{X + X,X + X) = ^i{[X + X,X + X]) 

= ^,i[x,x]) + f,{[x,x]) + ^,{[x,x]) + ^^{[x,x]). 

But [X,X] e and [X,X] e and from e p*(f)i) and a ^ it 

follows that 

li{[X,X]) = i^{[X,X])=0. 

Consequently 

^i{[x,x]) = ^Q^{x + x,x + x) > 0, 

and nix, X]) = implies that X = 0. 

Moreover, if n{[X,X]) = then Q^{X + X,X + X) from which it follows 
that X + X = 0. This means that iX G g, hence [f)Q, iX] C q. However, if 
i? G [)q is chosen such that a{H) ^ 0, then 

[H,iX] = i[H,X] = i^a{H)X = -a{H)X 

and this yields a contradiction because clearly —a{H)X ^ q. 

a 
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6.2 Application to real simple Lie superalgebras 

Let Q = (G, g) be a Lie supergroup sueh that G is connected and g is a 
real simple Lie superalgebra with nontrivial odd part. Assume that Q has 
nontrivial unitary representations. The goal of this section is use the necessary 
conditions obtained in Section 6.1 to obtain strong conditions on g. 

Since g is simple, G will be ★-reduced and Proposition 6.1.2(iv) implies 
that 00 contains a compactly embedded Cartan subalgebra. In particular, 
since complex simple Lie algebras do not have compactly embedded Cartan 
subalgebras, g should be a real form of a complex simple Lie superalgebra. 
However, as Theorem 6.2.1 below shows, for a large class of these real forms 
there are no nontrivial unitary representations. For simplicity, we exclude the 
real forms of exceptional cases G(3), F(4) and D(2|1,Q!). 

Theorem 6.2.1. If q is one of the following Lie superalgebras then Q does 
not have any nontrivial unitary representations. 

i) 5[(m|n,ffi) where m > 2 or n>2. 

ii) 5u{p,q\r,s) where p,q,r, s > 0. 

iii) su*(2p, 2q) where p,q > and p + q > 2. 

iv) P^{iTi) where m> 1. 

v) usp(to) where m> 1. 

vi) osp* (m|p, q) where p,q,m > 0. 

vii) osp{p, q\2n) where p,q,n> 0. 

viii) Real forms o/P(n), n> I. 

ix) psq(n, R) where n> 2, psq*(n) where n> 2, and 
psq{p, q), where p,q> 0. 

x) Real forms of'W{n), S{n), andS{n). 

xi) H(p, q) where p + q > 4. 

Proof. Throught the proof, for every n we denote the n x n identity matrix 
by I„, and set 

" i„l 

-In 



I. 



Ip 

-I„ 



and 3r, 



(i) Since [gYifli] — sl(m,M) ©sl(n,M) has no compactly embedded Cartan 
subalgebra, this follows from Proposition 6.1.2(iv). 

(ii) In the standard realization of sl{p + q\r + s, C) as quadratic matrices 
of size p + q + r + s, su{p, q\r, s) can be described as 



A B 
CD 



&s[{p + q\r + s,C) 



— T A*^ 



il 



A B 
CD 



Suppose, on the contrary, that Q is ★-reduced. Proposition 6.1.2(iii) implies 
that the diagonal matrices in su{p, q\r, s) constitute a Cartan subalgebra of 
su{p, q\r, s)q which is compactly embedded in su{p, q\r, s). Let fi be chosen as 
in Propostion 6.1.3. For every a <r and b < p, the matrix 
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Xa,b = 





E„,6 



is a root vector. Let r denote the complex conjugation corresponding to the 
above reahzation of su{p, q\r, s). One can easily check that 



iEfc,, 




Set Ha^b = [Xa,bjT{Xa,b)]- It is easily checked that 

Ha,b 



tEb,b 

iEa,a 



For a and b there are three other possibilities to consider. If a < r and b > p, 
or if a > r and b < p, then 



Ha,b = 

and if a > r and b > p then 

Ha,b - 





-lE„a 



i'Eb,b 
iE„,_ 



Proposition 6.1.3 implies that /u(iJa,6) > for every 1 < a < p + q and every 
1 < 6 < r+s. However, from the assumption that p, q, r, and s are all positive, 
it follows that the zero matrix lies in the convex hull of the HafiS, which is 
a contradiction. Therefore Q cannot be ★-reduced. 

(iii) Note that su*(2p|2g)g ~ su*(2p) ©su*(2g). The maximal compact 
subalgebra of su*(2n) is sp(n), which has rank n. The rank of the complex- 
ification of 5U*(2n), which is sl(2n,C), is 2n — 1. If n > 1, then 2n — 1 > n 
implies that su* (2n) does not have a compactly embedded Cartan subalgebra. 
Now use Proposition 6.1.2(1) and Lemma 5.2.1. 

(iv) This Lie superalgebra is a quotient of q(m) by its center, where q(m) 
is defined in the standard realization of s[(m|m, C) by 



q(m) = 



A B 
CD 



€ sl{m\m, 





D C 




'A B 






B A 




CD 


} 



One can now use Proposition 6.1.2(iv) because q(TO)Q = C) ©M contains 
no compactly embedded Cartan subalgebra. 

(v) This Lie superalgebra is a quotient of up(m) by its center, where up(m) 
is defined in the standard realization of sl(m|m, C) by 



A B 
CD 



e si{m\m,C) 





-D* 


B* ' 




'A B 






-C* 


-A* 




CD 


} 
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This implies that up(TO)g = sl(m, C) © M. Since this Lie algebra has no com- 
pactly embedded Cartan subalgebra, the assertion follows from Proposition 
6.1.2(iv). 

(vi) From Section 5.3 it follows that osp*{m\p, q)^ ~ so*(m) ©sp(p, q) has 
pointed generating invariant cones if and only if p = or g = 0. One can now 
use Proposition 6.1.2(i). 

(vii) The argument for this case is quite similar to the one given for 
su{p,q\r,s), i.e., the idea is to find root vectors € Qj'" such that the 
convex hull of the [X(j,T(Xa)]'s contains the origin. The details are left to 
the reader, but it may be helpful to illustrate how one can find the root 
vectors. The complex simple Lie superalgebra osp(m|2n, C) can be realized 
inside sl(rn|2n, C) as 



05p(m|2n,C) 



A B 
C D 



-A* — C*Jn 



A B 
CD 



If p and q are nonnegativc integers satisfying p + q = m then osp{p, q\2n) is 
the set of fixed points of the map 



T : osp(rn|2n, ( 



defined by 



'A B 




C D 


H 



osp(m|2n, C) 



D 



Moreover, osp(p, ql^n)-^ ~ so{p, q) © sp(2n, M) consists of block diagonal ma- 
trices, i.e., matrices for which B and C are zero. 

Assume that osp{p,q\2n) is ★-reduced. Then the span of 



{Ej,p+i-j - Ep+i_j,j 1 1 < i < 



and 



{Ep+j,P+9+l-J ~ '^P+q+'^-hP+j I 1 < i < L|j} 

is a compactly embedded Cartan subalgebra of so(p, g), and the span of 

{^p+q+j,p+q+n+j ~ '^p+q+n+j,p+q+j \ ^ ^ 3 — ^} 

is a compactly embedded Cartan subalgebra of sp(2n,IR). 

Fix 1 < 6 < n. For every a < p we can obtain two root vectors as follows. 
If we set 

Ba,b = Ea,6 + i^a,b+n + i^p+l-afi — Ep+l_a,6+Ti 

and 

Ca,b = —'i^b,a + Efc^p+i-a + Eb_|_„_a + iEb+n,p+l-a, 

then the matrix 



Ba^b 

Ca,b 
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Xa,b 

is a root vector, and Hafi = [^a,6, T{Xa,b)] is given by 

Ha,b 
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Aa,b 

D„b 



where Aafi = 2Ea^p+i_a - 2Ep_,_i_a_a and D^^b = -'^'E'b,b+n + 2E6+„,(,. Simi- 
larly, setting setting 



Ba,b = Ea,6 — iEa,b+n + i^p+l-a,b + ^p+l-a,b+n 



and 



Ca,b — 'Efc.o, — ^b,p+l-a + ^b+n,a + iEib+n,p+l-a 

yields another root vector Xa^b, and in this case for the corresponding Ha,b 
we have 

and 

Da.b = — 2E(,^6_|_„ + 2E(,+„^(,. 
Moreover, when p is odd, setting 



and 



yields a root vector X^p+i-^ ^, and -ffj-P+i-i ^ is given by 



and 



-C|-p+ii f, = — 2Eb^6+„ + 2E6+„_b. 



The case p < a <p + q is similar. 

(viii) Follows from Proposition 2.4.2, as the root system of P(n) is not 

symmetric. 

(ix) For psq(n,]R) and psq*(n), use Proposition 6.1.2(iv) and the fact that 

psq(n, M)q ~ s[(n, M) and psq*(n)Q ~ su*(n). 

For psq{p,q) and p,q > 0, we observe that it is a quotient of the subsu- 
peralgebra Q of sl{p + q\p + q, C) given by 



Q=sq{p,q) 



A B 
B A 



— T 4*7 'i^ R*T 

P:Q ^-^PyQ-^ P:1 

,T R*T — T /4*T 

p,Q P^Q P^Q P^Q 



A B 
B A 
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Let ^ G C be a squareroot of i. Then the maps 
and 



A 
A 



C"^^' 
C^B 



are linear isomorphisms. Note that 6q = u(p) © u(g) is a maximal compactly 
embedded subalgebra of 0q . Its center is 



^(%) 



and Qq is quasihermitian. The projection pj : u(p, 

by 

■itr(a)Ip 
itr(d)I, 



^{^q) is simply given 



Pi 



a b 




b* d 





Let C C 5q be the closed convex cone generated by [X, X], X E Qj. Since 
5q is quasihermitian, Lemma 5.3.2 implies that p^{C) = C fl ^(%). 
Next we observe that 



For B 



for every B G u{p, q) . 



C"^s' 


2 







C-^B 







-iB\ 



a b 
b* d 



so that 



we have 



B' 



a b 


2 


p* d 




= —i 


ritr( 



a'^ + bb* ab + bd 
b*a + ab* b*b + 



(2 + bb* 






■tr(6*6 + (i2) 



Applying this to positive multiples of matrices where only the a, b or d- 
component is non-zero, we see that the closed convex cone p^{C) contains 
the elements 



Zi = 



Hp 




Z2 = 







and Z3 = 



Hl„ 
izL 

q 9 



This implies that p^{C) = Jr(%) C C. 

We conclude that ^(flg) = «Mp+g ^ C, so that C = (g-^) (B Ci, where 
Ci = C n [flo)0o] is a non-pointed non-zero invariant closed convex cone in 
a simple Lie algebra isomorphic to su{p,q). This leads to Ci = [flo'flo]- 
conclude that C = Qq and the same holds also for the quotient psq{p, q). 
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(x) Follows from Proposition 2.4.2, as the root systems of these complex 
simple Lie supcralgebras are not symmetric (sec [Pe98, App. A]). 

(xi) Suppose, on the contrary, that G is *-reduced. Proposition 6.1.2(1) and 
Lemma 5.1.1 imply that every abelian ideal of g = 11{p,q) lies in its center. 
The standard Z-grading of'H.{p + q) (see [Ka77, Prop. 3.3.6]) yields a grading 
of H(p + g)o, i.e., 

H(p + g)o = H(p + q)^°^ e H(p + q)^f e • • ■ H(p + q)^-'^ 

where A;=p + g' — 3ifp + g'is odd and k = p + q — A otherwise. This grading is 

consistent with the real form H(p, q)^. Since H(p, q)!^^ is an abelian ideal of 

H(p, q)^, it should lie in the center of H(p, q)^. It follows that 'tl{p + q)'^'' lies 
in the center of H(p+q)g. However, this is impossible because it is known (see 
[Ka77, Prop. 3.3.6]) that tlip + q)^^ ~ so(p + q,C) and the representation of 

nip + q)^^ on nip + q)^^^ is isomorphic to a''+^Cp+i, from which it follows 
that 

[H(p + g)f ,H(p + g)f)]^{0}. □ 

Remark 6.2.2. In classical cases, Theorem 6.2.1 can be viewed as a converse to 
the classification of highest weight modules obtained in [Ja94] . From Theorem 
6.2.1 it also follows that for the nonclassical cases, unitary representations 

are rare. 

Remark 6.2.3. The results of [Ja94] imply that real forms of A{rn\m) do not 
have any unitarizable highest weight modules. However, A.(m\m) is a quotient 
of sI(to|to, C), and there exist unitarizable modules of su(p, m— p|m, 0) which 
do not factor to the simple quotient. For instance, the standard representation 
is a finite dimensional unitarizable module of su(m, 0|m, 0) with this property. 



6.3 Application to real semisimple Lie superalgebras 

Although real semisimple Lie superalgebras may have a complicated struc- 
ture, those which have faithful unitary representations are relatively easy to 
describe. 

Given a finite dimensional real Lie superalgebra g, let us call it ★-reduced 
if there exists a ^-reduced Lie supergroup Q = {G,g). 

Theorem 6.3.1. Let Q = {G,q) be a -k-reduced Lie supergroup. If g is a 

real semisimple Lie superalgebra then there exist -k-reduced real simple Lie 
superalgebras Si, ... ,Sk such that 



Si e • • • esfe C C DerR(si) © • • • © DerK(Sfc). 
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Proof. We use the description of g given in Theorem 2.5.1. First note that for 

every i we have — 0. To sec this, suppose on the contrary that > for 
some i, and let . . . , be the standard generators of AK.(nj). For every 
nonzero X e (Si)o have X G (s,)- and 

[X(^a,x®a] = 0. 

Proposition 6.1.1 impUes that X ® hes in the kernel of every unitary rep- 
resentation of Q, which is a contradiction. 

From the fact that all of the rij, 1 < i < fc, are zero it follows that 

si e • • • e Sfe C C DerKi (si) e • ■ • DerK^ [sk) 

and from C g it follows that every Sj is ^-reduced. □ 



6.4 Application to nilpotent Lie supergroups 

Another interesting by-product of the results of Section 6.1 is the following 
statement about unitary representations of nilpotent Lie supergroups. (A Lie 
supergroup Q = (G, g) is called nilpotent if g is nilpotent.) 

Theorem 6.4.1. // (tt, , ^) is a unitary representation of a nilpotent Lie 
supergroup (G,g) then p'^Hqj, [gT^flll]) = {0}- 

Proof. By passing to a quotient one can see that it suffices to show that if 
(G,g) is nilpotent and ★-reduced then [gYjlflT'Sl]] ~ {^l- Without loss of 
generality one can assume that gg — [gj, g-]- By Proposition 6.1.2(iv) there 
exists a Cartan subalgebra f)g of gg which is compactly embedded in g. As gg is 
nilpotent, we have gg = ()g. Proposition 2.4.1 implies that gg acts semisimply 
on g. Nevertheless, since g is nilpotent, for every X G Qq the linear map 

ad(X) : g -> g 

is nilpotent. It follows that [gg, g] = {0}. In particular, [gj, [gj, gj]] = {0}. □ 
7 Highest weight theory 

For Lie supergroups whose Lie algebra g is generated by its odd part, we 
analyse in this section the structure of the irreducible unitary representations. 
The main result is Theorem 7.3.2 which asserts that this structure is quite 
similar to the structure of highest weight modules. Here it is generated by an 
irreducible representation of a Clifford Lie superalgebra and not simply by a 
an eigenvector. 
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1.1 A Frechet space of analytic vectors 



Let G be a connected Lie group with Lie algebra g. Let t C g be a com- 
pactly embedded Cartan subalgebra, and T = exp(t) be the corresponding 
subgroup of G. Then carries a norm || • || which is invariant under Ad(T). 
In particular, for each r > 0, the open ball Br = {X G Q^: \\X\\ < r} is an 
open subset which is invariant under Ad(r). 

Let (tt, Jf) be a unitary representation of G. A smooth vector v e Jif°° 
is analytic if and only if there exists an r > such that the power series 

oo 

f,:Br^Jf, MX) = y2-dn{Xrv (8) 

n=0 

defines a holomorphic function on Br- In fact, if the series (8) converges on 
some Br, then it defines a holomorphic function, and the theory of analytic 
vectors for unitary one-parameter groups implies that fv{X) = Tr{exp{X))v 
for every X G Br fl g. Therefore the orbit map of v is analytic. 

If the series (8) converges on Br, it converges uniformly on Bg for every 
s < r ([BoSi71, Prop. 4.1]). This means that the seminorms 

Qr^iv) = sup{||d7r(X)"t;|| I ||X|| <l,Xeg} 

satisfy 

— g„(v) < 00 for every s < r. 

n=0 

Note that the seminorms (?„ define the topology of Jif°° (cf. [NelO, Prop. 4.6]). 

For every r > 0, let J^^'^ denote the set of all analytic vectors for which 
(8) converges on Br, so that 

j^'^ =\J jf"^''-. 

r>0 

If u e Jif"'"' and s<r, set 



n=0 

and note that this is a norm on J^'^'^. 



Lemma 7.1.1. The norms Ps, s < r, turn into a Frechet space. 

Proof. Since Ps < pt foi s < t < r, the topology on J^'^'^ is defined by the 
sequence of seminorms {psr^)nen foi' sequence (s„) with s„ r. Therefore 
Jtf'^''^ is metrizable and we have to show that it is complete. 
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If (Vn) is a Cauchy sequence in Jif"''^ then for every s < r the sequence 
: Bj, Jif of holomorphic functions converges uniformly on each Bg to 
some function f : B^. ^ , which imphes that / is holomorphic. 

Let V = /(O). Then, for each X G q and k G N, d7r(X)'^t;„ is a Cauchy 
sequence in J^. This implies that v G J^°° with d7r(X)*u„ dTr{X)''v for 
every X G Q and k gN ([BoSi71, Prop. 3.1]). Therefore / = on Br, and 
this means that v G Jf"'^ with ?;„ — > w in the topology of M'^''^ . □ 

Lemma 7.1.2. Ij K C_ G is a subgroup leaving the norm, \\-\\ on invariant, 
then the norms Ps, s <r, on Jif^'^ are K -invariant and the action of K on 
J^'^'^ is continuous. In particular, the action of K on J'^'^'^ integrates to a 
representation of the convolution algebra L^(K) on Jif"'"^ . 

Proof. Since K preserves the defining family of norms, continuity of the in- 
action on Jff"''^ follows if we show that all orbit maps are continuous at Ik, 
where Ik denotes the identity element of K. Let v G Jif"'^ and suppose that 
km in K. Then 

Ps{T^{km)v -v) = 22 —'ln{'n:{km)v - v) 

71=0 

and 

qn{T^{km)v -v) < (7„ (7r(fc„ + g„(w) = 2g„(u). 

Since K acts continuously on , qn{T^{km)v — v) ^ for every n e N, 
and since Ps{v) < oo, the Dominated Convergence Theorem implies that 

Ps{TT{kn)v - v) 0. 

The fact that J^'^''^ is complete implies that it can be considered as a 
subspace of the product space ns<r^' where % denotes the completion 
of with respect to the norm p,,. We thus obtain continuous isometric 

representations of K on the Banach spaces fs, which leads by integration to 
representations of L^{K) on these spaces (see [HR70, (40.26)]). Finally, since 
^W:'- Q Y[.,<r ^ closed by completeness (Lemma 7.1.1) and i^-invariant, 
it is also invariant under L^{K). □ 

From now on assume that r is small enoujh such that the exponential 
function of the simply connected Lie group G'^with Lie algebra maps 
Br diffeomorphically onto an open subset of G"'. For every X G the 
corresponding left and right invariant vector fields define differential operators 
on exp(i?^) by 



{L.f)ig) = I 



/(ffexp(iX)) and {Rxf){g) = ^^ 

t=0 



/(exp(tX)ff). 

t=o 



Define similar operators L*^ and R*x on Br by 



L*x{foexp\Br) = {Lxf)oexp\B^ and i?3c(/oexp |bJ = (i?jc/)oexp 
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One can see that 

L*xfv = fd7,{x)v and R*xfv = d7r(X) o /„. (9) 

If 9{ot{Br, M") denotes the Frechet space of holomorphic Jf- valued functions 
on Br, then the subspace ^ol{Br, Jf")^ defined by 

:Ho[{Br,^Y = {/ e ^oC{Br,Jf) I R*xf = dTr{X) o f for every X e q}, 

is a closed subspace, hence a Prechet space. Therefore the map 

evo : ^oC {Br, M'Y , / H- /(O) (10) 

is a continuous linear isomorphism onto Jf'^''", hence a topological isomor- 
phism by the Open Mapping Theorem (see [Ru73, Thm. 2.11]). 
This implies in particular that 

Lemma 7.1.3. The subspace Jif^'^ C Jf is invariant under '^/{gp). 



7.2 Spectral theory for analytic vectors 

We have already seen in Lemma 7.1.2 that if {tt,J^) is a unitary repre- 
sentation of G then the subspaces J^'^'^ are invariant under the action of 
the convolution algebras of certain subgroups K C G. As a consequence, 
we shall now derive that elements of spectral subspaces of certain unitary 
one-parameter groups can be approximated by analytic vectors. 

We begin by a lemma about the relation between one-parameter groups 
and spectral measures. Let 03 (R) denote the space of Borel measurabe func- 
tions on R and ^{M.) denote the Schwartz space of M.. 

Lemma 7.2.1. Let 7 : M — > U(J^) be a unitary representation of the addi- 
tive group ofM. and A = A* = —{^'{0) be its self-adjoint generator, so that 
7(i) = e^*^ in terms of measurable functional calculus. Then the following 
assertions hold. 

(i) For each f G L^(M,C), we have j{f) = f{A), where 

f{x)= [ e'-yf{y)dy 

is the Fourier transform of f . 

(ii) Let P : *8(M) — > C{.y/f) be the unique spectral measure with A ~ P(idR). 
Then for every closed subset E CR. the condition v € P{E)J^ is equivalent 
to -f{f)v = for every f G ^(M) with /j^ = 0. 
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Proof. Since the unitary representation (7, is a direct sum of cyclic rep- 
resentations, it suffices to prove the assertions for cyclic representations. Ev- 
ery cyclic representation of R is equivalent to the representation on some 
space J>f = I/^(M, /u), where /x is a Borel probability measure on M and 



hit)Oix) = e'*^^(x) (see [NeOO, Thm. Vl.l.f f]). 

(i) This means that {A^){x) = x£,{x), so that J{A)^(x) = f{x)^{x). For 
every / S L^(IR, C) the equalities 



hold in the space Jf = L^(M, fi). 

(ii) In terms of functional calculus, we have P{E) = xe{A), where xe is 
the characteristic function of E. If Z]^ = 0, then Part (i) and the fact that 
fXE = imply that 



Conversely, suppose that v £ satisfies j{f)v = for every / e J^{M.) 
with ,/|^ = 0. If w ^ P{E)Jf, then P{E'')v ^ 0, and since E" is open and a 
countable union of compact subsets, there exists a compact subset B C E'^ 
with P{B)v ^ 0. Let V e C^°°(M) be such that V'|b = 1 and supp(V') C E". 
Then 

^ P{B)v = xb{A)v ^ ixB ■ i^){A)v = xb{A)^P{A)v 

implies that ^jJ{A)v 7^ 0. Since the Fourier transform defines a bijection 
^(R) y{R) ([Ru73]), there exists an / e with f = tp. Then 

l{f)v = I{Ay) = il>{A)v ^ 0, contradicting our assumption. This implies 

that V e P{E).yf. □ 

Proposition 7.2.2. Let (tt, ) be a unitary representation of the Lie group 
G and X e g such that the group e'*^'^^''^) preserves a norm \\ ■ \\ on . 
If P : *B(IR) — >■ C{y^) is the spectral measure of the unitary one-parameter 
group TTx{t) = 7r(exp(tX)) then for every open subset E C M. the subspace 

lp{E),J>f)r\.y^"^^'" is dense in P{E),y^'^^'" . 

Proof. On M"^'^ we consider the Frechet topology defined by the seminorms 
(Ps)s<r in Lemma 7.1.1. Applying Lemma 7.1.2 to K = exp(MX) implies that 

all of these seminorms are invariant under ttx (IR) and ttx defines a continuous 
representation of M on which integrates to a representation 



(7(/)0(a;)= / f{t)e'*^^{x)dt = f{x)^{x) 



= (/• xe){A) = f{A)xE{A) = i{f)P{E). 



Tix ■■ {L\R,C),*) ^ Endc(Jr'^''') 



of the convolution algebra that is given by 
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This essentially means that the operators nxif) of the integrated represen- 
tation L-'^(R) C{,y^) preserve the subspace 

Next we write the open set E as the union of the compact subsets 

En :=^t&E \t\ < n,d\si{t,E'') > i| 

and observe that [j^P{En)Jf° is dense in P{E)Jf. For every n, there exists 
a compactly supported function /i„ e C^(R, M) such that supp(/i„) C E, 
0<h„<l,andh„\j^ = 1. Let /„ G y{R) with /„ = /i„. Then 

^Xifn) - fni-lMX)) = hn{-idTT{X)) 

and consequently 

P{En)Jf C nxifn)^ C P{E)Jif. 

Therefore the subspace of Jf"^''' is contained in P{E)M'. If 

w = P{E)v for some v e then 

nx{fn)w = nx{fn)P{E)v = nx{fn)v G Jf""'"- 

and 

\\^x{fn)u, - wf = \\hni-idn{X))w ^wf < \\PiE\En)wf ^ 

from which it follows that TTx{fn)'w w. □ 

Proposition 7.2.3. If Y € satisfies [X,Y] = i^Y then for every open 
subset E CM. the spectral measure of wx satisfies 

dTT{Y){P{E)J^ n C P{E + n)Jif. (11) 

Proof. To verify this relation, we first observe that 

nx{t)dTr{Y)v = dn{e*^'^^ {Y))nx{t)v = e'*''dn{Y)nx{t)v 

for every v G Jf°°. For / G ^(K), the continuity of the map 

^(R)^jr°°, f^nx{f)v 

leads to 

nx{f)d7r{Y)v = d7r{Y) [ f{t)e'*''nx{t)v = dn{Y)nx{f ■ e^)v 

where ep(t) = e'*^. If u e P{E)J^ and / vanishes on E + fi then the function 
{^iJ.fT— /(m + ■) vanishes on E, and Lemma 7.2.1(ii) implies that TTx{f ■ 
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e^)t; = 0. Applying Lemma 7.2.1(ii) again, we derive that d'iT{Y)v S P{E + 



7.5 Application to irreducible unitary representations 
of Lie supergroups 

Let (tt, p'", J^) be an irreducible unitary representation of the Lie supergroup 
G = (G, g). Before we turn to the fine structure of such a representation, we 
verify that Lemma 7.1.3 generalizes to the super context. 

Lemma 7.3.1. The subspace Jf"'^ C Jf is invariant under ^{g^). 

Proof. In view of Lemma 7.1.3, it only remains to show that, for every Y G Qj 
and V € we have p'^{Y)v G JT'^'''. For every X e % n S^, we have the 

relation 

Tr{expX)p-^{Y)v = p"" {e'"^^Y)Tr{exp X)v = p"" {e'"^^Y)f^{X). (12) 
The complex bilinear map 

0^ X Jf°° Jf°° , {Z,v)^ p^ {Z)v 
is continuous by Lemma 4.2.4 and therefore holomorphic. Moreover, the map 

% ^ 01 > A e Y 

is holomorphic. Since compositions of holomorphic maps are holomorphic, it 
therefore suffices to show that fv{Br) C and that the map fv- Bj. — >■ 
°° is holomorphic. In fact, this implies that the map 

g^nBr^J^, X >-^TT{expX)p'^{Y)v 

extends holomorphically to Br, i.e., p'^{Y)v e 
We recall the topological isomorphism 

evo: y{oC{Br,Jfy / ^ /(O). 

By definition of 9{oC{Br, Jf)^, we have for each X £ g^ the relation 

d^(x)o/, = i?3,/„, 

showing in particular that dTT{X) o /„ : Br — > ^ is a holomorphic function. 
From the definition of the topology on J^°°, it therefore follows that /„ is 
holomorphic as a map Br — >■ Jif°°. □ 

The following theorem clarifies the key features of the ^-representation 
on Jif°°. 
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Theorem 7.3.2. Let (tt, p'^, Jf) be an irreducible unitary representation of 
the Lie supergroup Q = {G, g) which is -k-reduced and satisfies 

00 = [0T'0t]- 

Pick a regular element Xq e int{cone.{Q)) and let t = % © t- 6e the cor- 
responding Cartan subsuperalgebra of q (see Lemma 5.2.1 and Proposition 
2.3.1). Suppose that no root vanishes on Xq. Then the following assertions 
hold. 

(i) % is compactly embedded and = { a € A \ a{Xa) > } satisfies 
A\{{)] = A+U-A+. 

(ii) The space M'^ of i- finite elements in is an irreducible Q-module 
which is a t-Q-weight module and dense in Jif. 

(iii) The maximal eigenspace 'f of ip^ {Xq) is a,n irreducible finite dimen- 
sional t-module on which tg- acts by some weight X G i^. It generates the 
g-module * and all other i^-weights in this space are of the form 

A — miai - • • • — mkOik, aj e A'^, k eN, mi, . . . , mk G N U {0}. 

(iv) Two representations [ti, p^,J^) and {it' , j^') of Q are isomorphic 
if and only if the corresponding {-representations on Y and Y' are iso- 
morphic. 

Proof, (i) Proposition 6.1.2 implies that Cone{Q) is a pointed generating in- 
variant cone and has a Cartan subalgcbra ijj which is compactly embedded 
in 0. Then the corresponding Cartan supersubalgebra is given by its central- 
izer i = ^(t^). Pick a regular element Xq G %n int{Cone{Q)), so that /i+ 
satisfies A\{0} = A+i) - A+. 

(ii) Recall from (7) that ip'"{Xo) < 0. We want to prove the existence of 
an eigenvector of maximal eigenvalue for ip'^(Xo). Let 

S = min{a(Xo)|a e A+} 

and note that 6 > 0. Let P{[a,b]), a < b gR, denote the spectral projections 
of the selfadjoint operator ip^{Xo) and put 

A = sup(Spec(ip'^(Xo))) < 0. 

Since {'K,p'",^) is irreducible and the space of analytic vectors is 
dense, there exists an r > with M"^''^ ^ {0}. Then the invariance of 
under ^ (g"') (Lemma 7.3.1) implies that J/if'^''^ is dense in Jif. Hence Propo- 
sition 7.2.2 implies that, for every e > 0, the intersection 

P(]/i-£,/i])jf'n Jf"''' 

is dense in P(]/U — £,/i])Jf . In particular, it contains a non-zero vector vq. 
We then obtain with Proposition 7.2.3 for £ < 5 and a e 
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p(0C'").;o CP(]^,^[)^ = {0}. 
In view of the Poincare-Birkhoff Witt Theorem, this leads to 

Since t"' commutes with Iq, the subspace {^)vo is contained in 
P([/x — £,yLt])Jf , so that Proposition 7.2.3 yields 

'^{q^)vo C P(] - 00, - 5])^^ + P([m - £, M])Jf 

for every e > 0. As '^{q^)vq is dense in , we obtain for every e > the 
relation P([/x — = P({/ti}). Hence ip(Xo)wo = /^fo- Since is spanned 
by ad (Xq) -eigenvectors, the same holds for '^{q^), and hence for '^{q^)vq. 
This means that ip^{XQ) is diagonalizable. Repeating the same argument for 
other regular elements in t fl lnt{Cone{Q)) forming a basis of t, we conclude 
that p'^{t) is diagonalizable, i.e., that J^f is the orthogonal direct sum of 
weight spaces for t, rcsp., the corresponding group T. 

Let y = P{{ii})Jtf be the maximal eigenspace of ip^{Xo). Then Proposi- 
tion 7.2.2 applied to sets of the form E =]fi — e,p + e[ implies that Jif"''^ D Y 
is dense in . Further Y is T-invariant, hence an orthogonal direct sum of 
T-weight spaces. From Lemma 7.1.2, applied to K = T, we now derive that 
in each T-weight space 'f°'{T), the intersection with M"^'^ is dense. 

Let u„ G n be a T-eigenvector. From the density of ^(0'^)wc« = 

oi in we then derive as above that 

is dense in As (ij) is finite dimensional, this proves that ^ = i^" is 
finite dimensional and contained in 

Since all tg- weight spaces in are finite dimensional and '^(ty) is 

finite dimensional, we conclude that (0*^)1^ is a locally finite t-module 
with finite tQ-multiplicities. In view of the finite multiplicities, its density 
in M' leads to the equality ' = '^i/ {q'^)^- As this g-module consists of 
analytic vectors, its irreducibility follows from the irreducibility of the Q- 
representation on 

(iii) If y C is a non-zero t-submodule, then (t)l^' is dense in and 
orthogonal to the subspace Y" = {y')'^, which leads to Y" = {0}. Therefore 
the t-module Y is irreducible. All other assertions have already been verified 
above. 

(iv) Clearly, the equivalence of the ^-representations implies equivalence 

of the t- representations on Y and Y' . 

Suppose, conversely, that there exists a i-isomorphism (j): Y — )■ Y' . We 
consider the direct sum representation ^ = 'K®'H! oiQ^ for which 
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as fl-modules. Consider the g-submodule W C generated by the t- 
submodule 

r{(t,) = {{v, (j){v)) : V er} cr (Bf. 

Since r{(j)) is annihilated by 5+, the PBW Theorem imphes that 
It follows that 

wn(r©r') = r(^) 

is the maximal eigenspace for iXo on W. 

As W consists of analytic vectors, its closure is a proper G- invariant 
subspace of J(f, so that we obtain a unitary tj-representation on this space. 

If the two Q representations {-K,p",^) and {-k' are not equiva- 
lent, then Schur's Lemma implies that M' and M" are the only non-trivial 
^-invariant subspaces of =J^, contradicting the existence of W . □ 

Remark 7.3.3. (a) The preceding theorem suggests to call the ^-representation 

on Jif^ a highest weight representation because it is generalized by a weight 
space space of tg- which is an irreducible t-module, hence a (finite dimensional) 
irreducible module of the Clifford Lie superalgebra tj]. 

(b) Suppose that g is reduced with = [qj, Qj] . Let be a complex 
Hilbert space and S' C Jif a dense subspace on which we have a unitary 
representation {p,2>) of q in the sense that (i), (iii), (v) in Definition 4.2.1 
are satisfied. 

Suppose further that the action of tg on ^ is diagonalizable with finite 
dimensional weight spaces. Then the g-module ^ is semisimple, hence irre- 
ducible if it is generated by a tg- weight space V on which t acts irreducibly. 

The finite dimensionality of the tg-weight spaces on & also implies the 
semisimplicity of ^ as a flg-module. Hence, as ip{Xo) < 0, an argument as 
in the proof of Theorem 7.3.2 implies that each simple submodule of is a 
unitary highest weight module, hence integrable by [NeOO, Cor. XII. 2. 7]. We 
conclude in particular that the go-representation on ^ is integrable with S> 
consisting of analytic vectors. 



8 The orbit method and nilpotent Lie supergroups 

One of the most elegant and powerful ideas in the theory of unitary repre- 
sentations of Lie groups since the early stages of its development is the orbit 
method. The basic idea of the orbit method is to attach unitary represen- 
tations to special homogeneous symplectic manifolds, such as the coadjoint 
orbits, in a natural way. One of the goals of the orbit method is to obtain a 
concrete realization of the representation and to extract information about 
the representation (e.g., its distribution character) from this realization. 
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Recall that a Lie supergroup Q = [G, q) is called nilpotent when the Lie 
superalgebra g is nilpotent. In this article the orbit method is only studied 
for nilpotent Lie supergroups. It is known that among Lie groups, the orbit 
method works best for the class of nilpotent ones. For further reading on the 
subject of the orbit method, the reader is referred to [Ki04] and [VoOO]. 



8.1 Quantization and polarizing subalgebras 

All of the irreducible unitary representations of nilpotent Lie groups can be 
classified by the orbit method. Let G be a nilpotent real Lie group and g be 
its Lie algebra. For simplicity, G is assumed to be simply connected. In this 
case, there exists a bijective correspondence between coadjoint orbits (i.e., 
G-orbits in g*) and irreducible unitary representations of G. In some sense 
the correspondence is surprisingly simple. To construct a representation ttq 
of G which corresponds to a coadjoint orbit O C g*, one first chooses an 
element A € O and considers the skew symmetric form 



It can be shown that there exist maximal isotropic subspaces of f2\ which are 
also subalgebras of g. Such subalgebras are called polarizing subalgebras. For 
a given polarizing subalgebra m of g, one can consider the one dimensional 
representation of the subgroup M = exp(m) of G given by 



The unitary representation of G corresponding to O is ttq = Indj^^XA- Of 
course one needs to prove that the construction is independent of the choices 
of A and m, the representation ttq is irreducibk\ and the correspondence is 
bijective. These statements are proved in [Ki62]. Many other proofs have been 
found as well. 



8.2 Heisenberg-Clifford Lie supergroups 

Heisenberg groups play a distinguished role in the harmonic analysis of nilpo- 
tent Lie groups. Therefore it is natural to expect that the analogues of Heisen- 
berg groups in the category of Lie supergroups play a similar role in the 
representation theory of nilpotent Lie supergroups. These analogues, which 
deserve to be called Heisenberg-Clifford Lie supergroups, can be described as 
follows. Let (W, Q.) be a finite dimensional real super symplectic vector space. 
This means that W = Wff © Wy is endowed with a bilinear form 



f?A : X g M defined by f2x{X, Y) = X{[X, Y]). 



(13) 



X\im) = e 



iA(log(m)) fQj. 
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that satisfies the following properties. 

(i) n{\N^,\Nj) = n(y\jj,\N^) = {o}. 

(ii) The restriction of Q to Wq is a symplectic form. 

(iii) The restriction of Q to is a nondegenerate symmetric form. 

The Heisenberg Clifford Lie supergroup corresponding to (W, Q.) is the super 

Harish- Chandra pair (iJ™,f)™) where 

(i) (^^ = Wg © R and t)j = Wy (as vector spaces). 

(ii) for every X,Y G \N and every a,b e R, the superbracket of f)^ is 
defined by 

[{X,a),iY,b)] = {0,QiX,Y)). 

(iii) is the simply connected Lie group with Lie algebra f)^. 

When dim Wj = the Lie supergroup (H^ , 1)™) is purely even, i.e., it is 
a Lie group. In this case, it is usually called a Heisenberg Lie group. When 
dimWg = the Lie supergroup (ff^, f)^) is called a Clifford Lie supergroup. 

Irreducible unitary representations of Heisenberg Lie groups are quite easy 
to classify. One can use the orbit method of Section 8.1 to classify them, but 
their classification was known as a consequence of the Stone-von Neumann 
Theorem long before the orbit method was developed. The Stone-von Neu- 
mann Theorem implies that there exists a bijective correspondence between 
infinite dimensional irreducible unitary representations of a Heisenberg Lie 
group and nontrivial characters (i.e., one dimensional unitary representa- 
tions) of its center. 

For Heisenberg-Clifford Lie supergroups there is a similar classification 
of representations. Let (ir^p^^Jf) be an irreducible unitary representation 
of (if™,[)™). By a super version of Schur's Lemma, for every Z G ^(()™) 
the action of p"{Z) is via multiplication by a scalar Cp-n{Z). If Cp7r(Z) = 
for every Z G .^(t)™), then ,3^ is one dimensional, and essentially obtained 
from a unitary character of Wq. The irreducible unitary representations for 
which p'^{2f{i)^)) 7^ {0} are classified by the following statement (see [SalO, 
Thm. 5.2.1]). 

Theorem 8.2.1. LetS be the set of unitary equivalence classes of irreducible 
unitary representations (tt, p"", ^) of {H^ , ()™) for which p'^(^([)™)) {0}. 
Then S is nonempty if and only if the restriction of Q to Wy is (positive or 
negative) definite. Moreover, the map 

[{'K,p\Jf)\^Cp. 

yields a surjection from S onto the set of M.-linear functional 7 : 3f{i)^) — ?■ M 
which satisfy 

i-f{[X,X]) < for every O^X G\Nj. 
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When dim \Nj is odd the latter map is a bijection, and when dim is even 
it is two-to-one, and the two representations in the fiber are isomorphic via 
parity change. 

Every irreducible unitary representation of a Clifford Lie supergroup is fi- 
nite dimensional (see [SalO, Sec. 4.5]). In fact the theory of Clifford modules 
implies that the only possible values for the dimension of such a representa- 
tion are one or 



It will be seen below that Clifford Lie supergroups are used to define analogues 
of polarizing subalgebras for Lie supergroups. 



8.3 Polarizing systems and a construction 

In order to construct the irreducible unitary representations of a nilpotent 

Lie supergroup using the orbit method, first we need to generalize the no- 
tion of polarizing subalgebras. What makes the case of Lie supergroups more 
complicated than the case of Lie groups is the fact that irreducible unitary 
representations of nilpotent Lie supergroups are not necessarily induced from 
one dimensional representations. However, it will be seen that they are in- 
duced from certain finite dimensional representations which are obtained from 
representations of Clifford Lie supergroups. 

Let (G, g) be a Lie supergroup. Associated to every A G there exists a 
skew symmetric bilinear form Qx on Qq which is defined in (13). There is also 
a symmetric bilinear form 



associated to A, which is defined by Q\{X,Y) = X{[X,Y]). 

Definition 8.3.1. Let Q = (G, g) be a nilpotent Lie supergroup. A polarizing 
system in (G, g) is a pair {A4, A) satisfying the following properties. 

(i) A G g^ and Q.\ is a positive scmidcfinitc form. 

(ii) M = (M, m) is a Lie subsupergroup of G and dim ruj = dim qj. 

(iii) mQ is a polarizing subalgebra of with respect to A, i.e., a subalgebra 
of Qq which is also a maximal isotropic subspace with respect to f2\. 

Given a polarizing system {A4,X), one can construct a unitary represen- 
tation of G as follows. Let 




(14) 



j =kerAerad(QA) 



(15) 



where rad(QA) denotes the radical of Q\. One can show that j is an ideal of m 
that corresponds to a Lie subsupergroup J = {J,j) of M, and the quotient 
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M/J is a Clifford Lie supergroup. Let 3f{m/]) denote the center of m/j. 

Since Q.\ is positive semidcfinitc, from Theorem 8.2.1 it follows that up to 
parity and unitary equivalence there exists a unique unitary representation 
{a,p'^ ,Jf) oiM/J such that for every Z e 2f{m/]), the operator p^iZ) acts 
via multiplication by i\{Z). Clearly {(T,p°',Ji^) can also be thought of as a 
representation of M, and one can consider the induced representation 

(7r,p^Jf)=Ind^(<7,p^Jr). (16) 



8.4 Existence of polarizing systems 

Throughout this section Q = (G, Q) will be a nilpotent Lie supergroup such 
that G is simply connected. 

It is natrual to ask for which A G such that Qa is positive semidefinite 
there exists a polarizing system {A4. X) in the sense of Definition 8.3.1. It 
turns out that for all such A the answer is affirmative. The latter statement 
can be proved as follows. Fix such a A € gi. Proving the existence of a 
polarizing system (Ai , A) amounts to showing that there exists a polarizing 
subalgebra mg of such that tUg 3 [gj, Qj] . Since [qj, Qj] is an ideal of the 
Lie algebra Qq and Qq is nilpotent, one can find a sequence of ideal of 0q such 
as 

{0} = C C . . . C iC^-l) C iC^) = [qj, Qj] C C . . . C = 0g 

where for every < s < r — 1, the codimension of i^*' in is equal to 

one. For every < s < r — 1, let 

be the skew symmetric form defined by 

4^\X,Y) = Xi[X,Y]) 

and let rad(f2^*^) denote the radical of . It is known that the subspace 
of flg defined by 

rad{n[^^) + ■ ■ ■ + rAd{n["^) 

is a polarizing subspace of Qq corresponding to A (see [CoGr90, Th. 1.3.5]). 
To prove that 

[01,01] Crad(r2«) + ---+rad(4^^) 

it suffices to show that rad(J7^'^'') = 1^*^^. This is where one needs the fact 
that Qx is positive semidefinite. The proof is by a backward induction on the 
dimension of ^. Details of the argument appear in [SalO, Sec. 6.3]. 
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8.5 A bijective correspondence 

Throughout this section Q = {G, q) will be a nilpotent Lie supergroup such 
that G is simply connected. 

One can check easily that the set 

^{G) = { A G I Qa is positive semidefinite } 

is an invariant cone in g^. Section 8.4 shows that for every A e ^^{G) one 
can find a polarizing system {M,X). Therefore the construction of Section 

8.3 yields a unitary representation {ttx, p"^ , y^x) of G which is given by (16). 
The main result of [SalO] can be stated as follows. 

Theorem 8.5.1. The map which takes a X € 3^{G) to the representation 
[ttx, p^^ , J^x) results in a bijective correspondence between G-orbits in ^{G) 
and irreducible unitary representations of G up to unitary equivalence and 
parity change. 

To prove Theorem 8.5.1 one needs to show that the construction given 
in Section 8.3 yields an irrcdiicible representation and is independent of the 
choice of A in a G-orbit or the polarizing system. One also has to show that if 
A and A' are not in the same G-orbit then inducing from polarizing systems 
{M, A) and {M', A') does not lead to representations which are identical up 
to parity or unitary equivalence. The proofs of all of these facts are given in 
[SalO, Sec. 6]. To some extent, the method of proof is similar to the original 
proof of the Lie group case in [Ki62], where induction on the dimension is 
used. In the Lie group case, what makes the inductive argument work is 
the existence of three dimensional Heisenberg subgroups in any nilpotent Lie 
group of dimension bigger than one with one dimensional center. For Lie 
supergroups a similar statement only holds under extra assumptions. The 
next proposition shows that it suffices to assume that the corresponding Lie 
superalgebra has no self-commuting odd elements. 

Proposition 8.5.2. Let G = (G, g) he as above. Assume that there are no 
nonzero X G Qj such that [X,X] = 0. //dim.^(g) = 1 then either G is a 
Clifford Lie supergroup, or it has a Heisenberg Lie subsupergroup of dimension 
(3|0). 

Using Proposition 6.1.1 one can pass to a quotient and reduce the analysis 
of the general case to the case where the assumptions of Proposition 8.5.2 are 
satisfied. Proposition 8.5.2 makes induction on the dimension of g possible. 

Although the proof of Theorem 8.5.1 is insipred by the methods and argu- 
ments in [Ki62] and [CoGr90] , one must tackle numerous additional analytic 
technical difficulties which emerge in the case of Lie supergroups. This is be- 
cause many facts in the theory of unitary representations of Lie supergroups 
are generally not as powerful as their analogues for Lie groups. For instance 
to prove that {nx,p'^'','y^\) is irreducible one cannot use Mackey theory and 
needs new ideas. 
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Let g = (G,0) be as in Section 8.5. For every A G .9>{g) let {ttx, p'^\ Jffi) 
be the representation of G associated to A in Section 8.5. As an application 
of Theorem 8.5.1 one can obtain a simple decomposition formula for the 
restriction of {n\, p^^ ,Jfx) to G. 

Recall that {nx,p^^ , Jfx) is induced from a polarizing system {M., A). Let 
m be the Lie superalgebra of M and j be defined as in (15). 

Corollary 8.6.1. The representation {ttxjJ^x) ofG decomposes into a direct 

sum, copies of the irreducible unitary representation of G which 

is associated to the coadjoint orbit containing X (in the sense of Section 8.1). 



9 Conclusion 

In this note we discussed irreducible unitary representations of Lie super- 
groups in some detail for the case where Q is either nilpotent or q is ★-reduced 
and satisfies Aq = [gj, 0t] • ^he overlap between these two classes is quite small 
because for any nilpotent Lie superalgebra satisfying the latter conditions 0q 
is central, so that it essentially is a Clifford-Lie superalgebra, possibly with a 
multidimensional center, and in this case the irreducible unitary representa- 
tions are the well-known spin representations. Precisely these representations 
occur as the t-modules on the highest weight space Y in the other case. 

Clearly, the condition of being ★-reduced is natural if one is interested in 
unitary representations. The requirement that Qq = [qj, Qj\ is more serious, as 
we have seen in the nilpotent case. In general one can consider the ideal Qc = 
[ST' 0t] ® At ^^"i '-"^'^ results show that the irreducible unitary representations 
of this ideal are highest weight representations. For nilpotent Lie supergroups, 
how to use them to parametrize the irreducible unitary representations of 
Q was explained in Section 8. It is conceivable that other larger classes of 
groups could be studied by combining tools from the Orbit Method, induction 
procedures and highest weight theory. 
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So/rad(So) 
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